Skip to content

Add Robot Framework E2E Tests for University Agent#21

Open
devin-ai-integration[bot] wants to merge 4 commits intomainfrom
devin/1760703523-robot-framework-e2e-tests
Open

Add Robot Framework E2E Tests for University Agent#21
devin-ai-integration[bot] wants to merge 4 commits intomainfrom
devin/1760703523-robot-framework-e2e-tests

Conversation

@devin-ai-integration
Copy link

@devin-ai-integration devin-ai-integration bot commented Oct 17, 2025

Description

This PR implements comprehensive end-to-end tests using Robot Framework for the University Agent, covering both the FastAPI REST API and Streamlit UI. The tests validate agent functionality including university searches, error handling, UI interactions, and service lifecycle management.

Link to Devin run: https://app.devin.ai/sessions/e663b31a138447dc9613ba3584e6cd55
Requested by: @thepollari

Changes

Test Infrastructure

  • Created tests/e2e/robot/ directory structure with 7 new files
  • api_tests.robot - 7 test cases covering API endpoints (health checks, country/name searches, error handling, timeouts)
  • ui_tests.robot - 9 test cases covering Streamlit UI (page loading, button interactions, chat input, status checks)
  • resources/keywords.robot - Reusable keywords for service management (start/stop agent & Streamlit, query sending, validation)
  • resources/variables.robot - Common variables (URLs, timeouts, ports)
  • fixtures/test_data.json - Test data with valid/invalid queries
  • README.md - Comprehensive documentation on running tests, troubleshooting, and test coverage

Dependencies

  • Added robotframework>=7.1.1, robotframework-browser>=19.10.1, robotframework-requests>=0.9.7 to pyproject.toml
  • Updated uv.lock with new dependencies and transitive packages

CI/CD Integration

  • Added robot-tests job to .github/workflows/check.yaml
  • Job installs dependencies, initializes Browser Library, runs tests, and uploads test artifacts (log.html, report.html, output.xml)
  • Requires GEMINI_API_KEY secret to be configured

Build Tools

  • Added robot-tests target to Makefile for local test execution

Type of Change

  • New feature
  • Documentation

Test Coverage

API Tests (7 cases):

  • Agent endpoint health check
  • Search by country (Finland) with validation
  • Search by name (Aalto University) with validation
  • Empty query handling
  • Response structure validation
  • Connection error handling
  • Timeout verification (<30s)

UI Tests (9 cases):

  • Streamlit page loads at http://localhost:8502
  • Page title displays correctly
  • "Check Agent Status" button functionality
  • All 5 example query buttons present
  • Example button click functionality
  • Chat input accepts queries
  • Agent responses display in chat
  • Clear conversation button
  • Agent URL configuration in sidebar

Additional Comments

⚠️ Important for Reviewers

  1. CI Secret Required - This PR assumes GEMINI_API_KEY is configured in GitHub Secrets. Please verify this secret exists before merging.

  2. Not Fully Tested - Due to environment limitations (missing browser system dependencies locally), these tests were only validated via Robot Framework dry runs to verify syntax. Actual test execution has not been performed. The tests should work in CI but will need monitoring on first run.

  3. Service Management - Tests use subprocess-based service lifecycle management (start agent on port 8001, Streamlit on port 8502). Hard-coded sleep times (10s for agent, 5s for Streamlit) may need adjustment based on CI performance.

  4. External API Dependency - Tests use the real universities.hipolabs.com API rather than mocks (Task 7 was optional). Tests may fail if the external API is down.

  5. Browser System Dependencies - The CI job runs rfbrowser init which requires system libraries. Standard Ubuntu GitHub Actions runners should have these, but this hasn't been verified.

Human Review Checklist

  • Verify GEMINI_API_KEY secret is configured in GitHub repository settings
  • Check if 10s agent startup wait and 5s Streamlit startup wait are sufficient
  • Review service management approach (subprocess-based) for robustness
  • Validate that all 8 tasks from original requirements are implemented
  • Monitor first CI run to ensure tests actually pass
  • Consider whether external API dependency is acceptable or if mocking should be added

Implementation Notes

All 8 tasks from the requirements were implemented:

  1. ✅ Test infrastructure and directory structure
  2. ✅ API test suite with 7 test cases
  3. ✅ UI test suite with 9 test cases (updated to 9 based on actual implementation)
  4. ✅ Reusable keywords for service management
  5. ✅ Suite setup/teardown with environment configuration
  6. ✅ Test data fixtures
  7. ⚠️ External API mocking (framework created but marked optional - using real API)
  8. ✅ CI/CD integration via GitHub Actions

@devin-ai-integration @thepollari
feat: Add Robot Framework E2E tests for University Agent
e4d5dee 
- Create test directory structure at tests/e2e/robot/
- Install Robot Framework dependencies (robotframework, robotframework-browser, robotframework-requests)
- Implement api_tests.robot with 7 test cases covering health checks, university search, error handling
- Implement ui_tests.robot with 9 test cases for Streamlit UI interactions
- Create reusable keywords for service lifecycle management (start/stop agent and Streamlit)
- Add suite setup/teardown with environment variable configuration
- Create test data fixtures with sample queries
- Integrate with CI/CD pipeline via check.yaml workflow
- Add Makefile target for running Robot tests
- Include comprehensive README documentation

All 8 tasks from requirements implemented (Task 7 API mocking is optional framework).
Tests verified with dry run - 16 tests total (7 API + 9 UI) all pass.

Co-Authored-By: jussi.pollari@merck.com <jussipol@yahoo.com>
@devin-ai-integration
Copy link
Author

devin-ai-integration bot commented Oct 17, 2025

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

devin-ai-integration bot and others added 3 commits October 17, 2025 12:25
@devin-ai-integration @thepollari
fix: Update robot-tests workflow to run from orchestrator directory
2450755 
The robot-tests CI job was failing because it tried to run 'uv run robot'
from the university directory, but Robot Framework dependencies are
installed in the orchestrator directory's pyproject.toml.

This fix changes the working directory to the orchestrator directory
and uses a relative path to reference the test files, matching the
approach that was successfully verified locally with dry run.

Co-Authored-By: jussi.pollari@merck.com <jussipol@yahoo.com>
@devin-ai-integration @thepollari
fix: Use relative path for agent service startup in Robot tests
47b0176 
Replace hardcoded absolute path '/home/ubuntu/repos/teal-agents/src/sk-agents'
with portable relative path using ${EXECDIR} variable.

The ${EXECDIR} variable points to the directory where robot command is executed
(orchestrator directory). From there, ../../../sk-agents correctly navigates to
the sk-agents directory regardless of the absolute filesystem path.

This fixes the FileNotFoundError that occurred in CI where the absolute path
differs from the local development environment.

Co-Authored-By: jussi.pollari@merck.com <jussipol@yahoo.com>
@devin-ai-integration @thepollari
fix: Use TA_API_KEY secret for GEMINI_API_KEY environment variable
253cf76 
The robot-tests CI job requires GEMINI_API_KEY to start the University Agent
service. Update the workflow to source this from the TA_API_KEY secret which
is already configured in the repository.

This fixes the 'Connection refused' error where the agent service failed to
start because GEMINI_API_KEY was not set (GEMINI_API_KEY secret doesn't exist
in the repo, but TA_API_KEY does).

Co-Authored-By: jussi.pollari@merck.com <jussipol@yahoo.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

0 participants

Comments